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Abstract 

This paper puts forward a set of criteria for ensuring trustworthiness in design research studies 
undertaken by Higher Degree Research (HDR) students. Design research is aimed at exploring 
educational problems and refining theory and practice by defining a pedagogical outcome and is a 
methodological approach often associated with the Learning Sciences. In this paper, how a solo HDR 
researcher can use a design research approach to maximise the benefits of the methodology without 
compromising the validity of the research design will be discussed. The criteria put forward to ensure 
reliability fall into two categories; trustworthiness and the research design. Overall, there is a perceived 
need and value for HDR students to contribute to the understandings of design studies which are 
guided by a trustworthy research design. 
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Introduction 

This paper offers perspectives on design research and the Higher Degree Research (HDR) experience. 
In research in the field of computer-supported learning the use of design research has gained a 
reputation as being the methodology of choice for the Learning Sciences and educational technology 
research (see, for example, (Barab, 2006; Barab & Squire, 2004; Edelson, 2002; Fishman, Marx, 
Blumenfeld, Krajcik, & Soloway, 2004; The Design-Based Research Collective, 2003). Design 
research lends itself to this field educational technology research as its underlying premise is to 
develop the design of artefacts, technological tools, and curriculum and to further an existing theory 
or develop new theories in naturalistic settings that can support and lead to an enhanced understanding 
of learning (Barab, Dodge, Thomas, Jackson, & Tuzun, 2007; Barab & Squire, 2004; Fishman, et al., 
2004). There has been a call for research that is focused on the design process itself at local level, as 
Schoenfeld (2009) explains that ‘the products of well conducted design experiments are improved 
interventions and improved understandings of the processes that result in their productiveness’, which 
are productive contributions to the research community. 

One characteristic of design research is the use of cross-disciplinary or multidisciplinary 
teams. The multidisciplinary teams are showcased in ongoing design studies, such as Quest Atlantis or 
River City, wherein the research often draws on the experience of several fields, such as the research 
staff, teachers, technological support staff, statisticians, the students and external stakeholders, such as 
government funding bodies (Ketelehut, et al., 2010; Barab, et al. 2007). This use of multidisciplinary 
research teams is seen as a strength of design research as a greater breadth of understanding can be 
brought into the research environment (Reeves, et al., 2005; Wang & Hannafin, 2005). Cobb et al. 
(2003) clarify that the size and type of research team depends on the purpose of the research and they 
explain that a one-to-one research design may be appropriate in a study wherein the researcher is 
conducting the teaching sessions. It is this, one-to-one research design that underpins the remainder of 
the paper. 
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In the HDR context, research students are often seen as working ‘solo’ on projects. That is to 
say, HDR candidates while not working in isolation may not he working collahoratively or are not 
part of a formalised research project, such as a grant firnded project. While they are working ‘solo’ on 
their own research, it can also he argued that HDR students, and all researchers, are part of a faculty 
or department and are thus able to draw from multi-disciplinary perspectives. It should he clarified 
here, that we are not stipulating that a HDR student that is conducting an investigation in the field 
must use this methodology. Rather, we argue that if a design research approach is the most 
appropriate methodology to investigate a research question then there are strategies that a solo 
researcher can use to ensure the integrity of the research design. In this paper, several measures are 
presented that a solo researcher can include in their design to fortify the validity and trustworthiness 
of their research design. 


Background 

Design Research 

‘Design research’ was put forward as a research approach that extends existing methods as a means to 
address the issue of linking theory and practice in educational research. The coining of the term 
‘design research’ is credited to Ann Brown in 1992 (Collins, Joseph, & Bielaczyc, 2004). Brown’s 
(1992) ‘design research’ converged qualitative and quantitative operations, collected multifaceted data 
and focused on in-depth proving of theory. Wang and Hannafin (2005) note that similar and 
sometimes interchanged terms, such as ‘design experiments’, ‘design-hased research’, ‘development 
research’, and ‘developmental research and formative research’, are often grouped with design 
research. While there are differences between the approaches many of the characteristics are shared. 
In this paper the term design research will be used. 

Design research is aimed at exploring educational problems and refining theory and practice by 
defining a pedagogical outcome and then focusing on how to create a learning environment that 
supports the outcome (Reeves, Herrington, & Oliver, 2005; Wang & Hannafin, 2005). According to 
Plomp (2007), design research is: 

Like all systematic educational and instructional design processes - therefore cyclical in 
character: analysis, design, evaluation and revision activities are iterated until a satislying 
balance between ideas (‘the intended’) and realisation has been achieved’ (p.l3). 

While design research has since been used across a range of educational settings, the basic 
concept of the research design is still very much apparent. Confrey (2006) explains that a design 
research study: 

Seeks to document what resources and prior knowledge the student brings to the task, how 
students and teachers interact, how records and inscriptions are created, how conceptions 
emerge and change, what resources are used, and how teaching is accomplished over the 
course of instruction, by studying student work, video records, and classroom assessments 
(p.135). 

Three ftindamental principles of design research are outlined by Reeves (2006): 

addressing complex problems in real contexts in collaboration with practitioners; integrating 
known and hypothetical design principles with technological advances to render plausible 
solutions to these complex problems; and conducting rigorous and reflective inquiry to test 
and refine innovative learning environments as well as to define new design principles (p.58). 
The ultimate goal of design research is to ‘build a stronger connection between educational 
research and real world problems’ (Amiel & Reeves, 2008, p. 34). The focus on iteration is not just to 
evaluate an innovation, for example a hardware or software, but rather to produce and refine design 
principles that can provide guidance for similar research studies or development endeavours (Amiel & 
Reeves, 2008). In Figure 1, a model of design research illustrates the iterative cycles which are 
characteristic of design research are part of the process of refining the solutions. It can be seen that 
that a design research approach supports development of design principles through a cycle of 
reflection, evaluation and refinement. 
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Ongoing cycles of reflection, iteration and re-design are allied with design research. The 
process of data collection is sustained over a period of several years. In studies into River City, a 
multi-user virtual environment used for scientific inquiry learning in high school, the evolution of the 
city and the design of the inquiry materials resulted from ongoing cycles of design and reflection 
(Ketelhut, Clarke, & Nelson, 2010; Ketelhut, Dede, Clarke, & Nelson, 2006). The reflection on the 
data gathered and subsequent re-design or addressing of issues is aimed at addressing the learning 
issue and achieving the research goals. As Wang and Hannafm (2005) suggest, in a design research 
study, data are analysed immediately, continuously and retrospectively and that part of this cycle of 
data collection involves stages such as a comprehensive literature review coupled with the systematic 
and purposeful implementation of research methods. This iterative process leads to the development 
of design principles, which are then reflected upon and evaluated through the refinement of the 
problem, solutions and methods. According to Amiel and Reeves (2008): 

The development of design principles will undergo a series of testing and refinement 
cycles. Data is collected systematically in order to re-defme the problems, possible 
solutions, and the principles that might best address them. As data is re-examined and 
reflected upon, new designs are created and implemented, producing a continuous 
cycle of design-reflection-design (p.35). 

What this means is that a research or research team systematically uses iterative cycles of design that 
inform subsequent design in a process of refining and redefinition of the design and the achievement 
of goals and the development or building on of valid theory (Edelson, 2002; Reeves, et ah, 2005; 

Wang & Hannafm, 2005). Hence, through the cycles of analysis, development, testing and 
refinement and reflection and evaluation, the principles and the solution implementation are 
revised and refined. 


Predictive research 



Design-based research 



Refinement of problems, solutions, methods, and design principles 


Figure 1. Model of Design-Based Research (Amiel & Reeves, 2008) 

In this respect, design research is often seen as a series of approaches, rather than a single 
approach that is aimed at the development of new theories and practices in naturalistic settings (Barab 
& Squire, 2004; The Design-Based Research Collective, 2003). The emphasis on naturalistic settings 
is emphasised in discourse on design research as the focus is on developing contextualised, but 
sharable, theories and cumulative design knowledge in classroom or learning environments. This is 
articulated by Schoenfeld (2009) who states that ‘properly constmcted, a design experiment consists 
of the creation of an instructional intervention on the basis of a local theory regarding the 
development of particular understandings’. Design research combines qualitative and quantitative data 
collection approaches, this combination of data collection strategies allows for a more robust 
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understanding of the learning environment (Bannan-Ritland, 2003; Brown, 1992; Fishman, et ah, 
2004; The Design-Based Research Collective, 2003; Wang & Ftannafin, 2005). 

Design research has been used across a range of educational settings this is due to in part to 
the adaptability of the design to classrooms, as Reimann (2010) explains ‘one of the main motivations 
behind design research is to make learning research more relevant for classroom practices’ (p.37). As 
such, design research tends to be adopted by researchers who are conducting studies in authentic 
classroom situations in order to generate theory and design relevant to a particular context. Design 
research is often at the convergence of design and theory and the design research framework supports 
traditional outcome-based evaluation and the importance of design (Edelson, 2002). The emphasis is 
not on refining education practice, but on addressing and dealing with theoretical issues and questions 
that arise (Collins, et al., 2004). Systematic evaluation of the consecutive research phases or iterations 
contributes to theory building (Plomp, 2007). 

While design research is often associated with the Learning Sciences, a field that is known for 
its utilisation of technology in education, the focus of a design research approach is on pedagogy and 
learning theories rather than on the development of technological tools and artefacts (Barab & Squire, 
2004; Wang & Ftannafin, 2005). While technology is often an important feature of research that 
adopts a design research approach, the learner and the teachers are still the focus. The use of authentic 
settings contribute to the research process and the legitimacy of the research is reflected in the 
acknowledgement of both the success and the failures, and if the authenticity of the environment is 
compromised as a result of the research, then this must too be documented and reflected upon in the 
data analysis (Fishman, Marx, Blumenfeld, Krajcik, & Soloway, 2004). 

Plomp (2007) outlines several dilemmas of design research. These are that the researcher is 
the designer and often the evaluator and implementor. Real world settings, such as the classroom, 
result in real world complications. There is also an issue of the adaptability. Each cycle of design 
research considers the findings of the preceding cycle; hence, the research design needs to be flexibly 
adaptive. One issue that is raised in critiques and literature on design research is how research 
undertaken at a local level can contribute to broader theories of learning and design (see, for example, 
Dede (2004)). As Barab and Squire (2004) note, it is easy to show learning gains at a local level, such 
as a classroom or school, but it is difficult to show the usefulness and consequenfiality of fhis work af 
a larger level. This adapfability is also question by Dede (2004), who finds fhat often the results of a 
design research activity are common sense, the large data set results in very general and under 
conceptualised findings. Dede (2004) also nofes fhat the cycles of iteration may incorporate numerous 
conditions for success that may affect the adoption of the design or theory at a large level. Scale and 
sustainability of the findings on broader levels are questioned by Fishman et al. (2004) and Barab et 
al. (2006) who point out that variables such as ease of adoption, sustainability of the design or theory 
and spread are dependent variables that needs to be considered in the research. This final issue is 
perhaps fhe mosf crucial factor fhat impacts upon a higher research student’s ability to undertake 
design research. Given that access to schools and recruiting participants is difficult, ft is unlikely that 
a solo student research will be able to show the usefulness of their design at a larger level. 

Schoenfeld (2009) outlines several areas where HDR students can contribute both to areas of 
research and in clarifying fhe design research process. He suggesfs fhaf HDR studenfs can (a) 
chronicle the design process and the piloting process, this could give rise to a range of design 
heuristics and descriptors of design principles that proved to be productive, (b) document professional 
pathways and essential skills for educational designers , (c) explore and identify the ways in the 
phases or aspects of the design process contributed to the final study or product, this may include top- 
down and checklists of ‘fail-safes’ for the design process, and (d) focus on a specific area of 
knowledge in the design process, such as design principles, design techniques or professional practice, 
and investigating the impact of these areas and how they work. As such, there is a perceived need and 
value for HDR students to contribute to the understandings of design studies which are guided by a 
trustworthy research design. 


Ensuring Trustworthiness in the Research Design 
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In order to ensure trustworthiness in a solo researcher design research study the inclusion of several 
criteria are proposed. These are based, in part, on Guha’s (1981) criteria that should he considered hy 
qualitative researchers in pursuit of a trustworthy study: dependability, credibility, transferability, 
confirmability. The criteria engaged in HDR, such as Mafumiko’s (2006) thesis on curriculum 
redesign and Squire’s (2004) thesis on game-based learning, to ensure reliability and trustworthiness 
are also drawn upon for research design. The criteria are outlined in Table 1. 


Table 1. Strategies to Support Reliability in Solo Research 


T rustworthiness 


Description 

Clarification 

Dependability 

Work repeated in same context 
with similar methods and 
participants the results would be 
similar 

Replication of phases with 
similar student cohorts 

Credibility 

To seek to ensure that a study 
measures or tests what is 
actually intended 

A study that supports prolonged 
exposure and persistent 
observation of the student 
group. Ongoing peer debriefing 

Transferability 

The extent to which the findings 
from one study can be applied 
to another 

Provide information on the 
duration of the study, the 
number of organisations and 
their locations, number of 
participants, data collection 
methods, number and length of 
data collection sessions 

Confinnability 

To seek to ensure that the 
findings of the research are the 
result of the ideas and 
experiences of the participants 
rather than the characteristics 
and preferences of the 
researcher 

Acknowledge the shortcomings 
in study’s methods and their 
potential effects on the research. 
The inclusion of an in-depth 
methodological description 

Design 


Description 

Clarification 

Micro-phases 

Series of iterative data 
collection phases that test and 
evaluate design. Each cycle in 
the study is a piece of research 
in itself 

Several tests of materials prior 
to main field test that examine a 
different aspect of the design or 
theory 

Expert groups 

The inclusion of several expert 
groups throughout the study to 
evaluate the materials and data 
collection instruments 

Engage with a mentor external 
to research of visiting scholars 
to review design. Submit 
research stages for peer review 
in conferences 

Different participant groups 

Test materials with a range of 
participants groups 

Engage groups, such as 
teachers, pre-service teachers. 


designers, in study prior to 
testing with students 
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Flexibly adaptive 


Ability to take on several roles 
without losing sight of the role 
of researcher 


Use of evolutionary planning 
framework which is responsive 
to field data and experiences as 
acceptable moments during the 
course of a study 


Trustworthiness 

The criteria to ensure trustworthiness presented in Table 1 are expanded on below. 

Dependability 

Dependability is linked to both credibility and confirmability (Lincoln & Guba, 1985). According to 
Shenton (2004), a researcher can establish dependability ‘if the work were repeated, in the same 
context, with the same methods and with the same participants, similar results would be obtained’ 
(p.71). Lienee each stage of the study should be informed by research on similar studies. In addition 
develop the consistency of the approach, the repetition of the phases is encouraged in investigation of 
‘all reasonable areas’ to ensure that early closure does not occur, thus reducing the impact of 
researcher bias (Lincoln & Guba, 1985). 

As reliability of the instruments is of great importance too, repeating the use of the survey 
instruments should ensure that the methods are reliable (Leedy & Ellis Ormrod, 2005; Punch, 1998). 
This testing of replication should be feasible and may allow for the reliability of the methods. 

Credibility 

The trustworthiness of design research study is intrinsically linked to credibility. Shenton (2004) 
explains that to establish credibility is to ‘seek to ensure that their study measures or tests what is 
actually intended’ (p.64). The term ‘credibility’ has been used in place of ‘internal validity’ and 
Lincoln and Guba (1985) propose five ways of ensuring credibility: prolonged engagement, persistent 
observation, triangulation, peer debriefing and negative case analysis. 

Prolonged engagement 

According to Lincoln and Guba (1985), prolonged engagement is crucial in helping to support the 
concept of credibility in qualitative research because it assists the researcher in testing for 
misinformation and building trust. Given the iterative nature of design research it is possible for a 
researcher to have prolonged engagement with a design. However, having prolonged engagement 
with a cohort of participants in a school or university context can be challenging. 

Persistent observation 

Persistent observation is the ongoing observation of participants in a study. Persistent observation 
allows the researcher to identify what is relevant to the study and what is not (Lincoln & Guba 1985). 
Through persistent observation, a solo researcher can also see how students and teachers function, 
which groups are motivated, which groups struggle, and how the teachers interact with the students. 

Triangulation 

Triangulation allows a researcher to view events from multiple perspectives. Lincoln and Guba (1985) 
outline several categories of friangulation, including triangulation by the use of multiple and different 
data, methods, investigators, and theories. 
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Peer debriefing 

Peer debriefing is a process in which the investigator discusses the investigation with peers. Through 
peer de-briefing a solo research can explore aspects of the research that may otherwise remain only 
implicit (Lincoln & Guba, 1985). Peer debriefing can encourage a solo researcher to search for biases, 
scrutinise their hypotheses and justification for their research, discuss the direction of their research 
and methodological design, and to explore their feelings and emotions towards their research so that 
they can assess how their experience might impact upon their interpretation of the data (Lincoln & 
Guba, 1985). 

Through the establishment of a rigorous peer debriefing process and open channels of 
discussion and communication with educators, a critical analysis of the observations and 
interpretations of the data can be achieved. As the facilitator of all of the sessions, solo researchers 
need to be cautious of teaching to the desired outcome and of being overly positive in their recording 
of observations and interpretation of the data. To maintain integrity, solo researcher should establish 
an ongoing process of peer debriefing with several senior researchers who could provide a critical 
perspective of the interpretation of the results as well as providing alternative interpretations of the 
data. By involving educators, such as school teachers and university lecturers, in the analysis of the 
results and by garnering their reflections and feedback, solo researchers can ensure that their research 
design is valid and viable. 

Peer debriefing can also include the presentation of stages of the research at both local and 
international conferences. The sharing of ideas can help HDR students not only in terms of the 
validation of their research and the development of trustworthiness, but can also help HDR students to 
gain a better understating of the field of technology-enhanced learning. 

Transferability 

Transferability is the extent to which the findings from one study can be applied to another (Shenton, 
2004). Lincoln and Guba (1985) argue that it is the responsibility of the investigator to ensure that 
sufficient contextual information about the fieldwork sites is provided to enable the reader to make 
such a transfer. Shenton (2004) proposes that the information relating to the following six issues 
should be provided in the research: 

a) the number of organisations taking part in the study and where they are based 

b) any restrictions in the type of people who contributed data 

c) the number of participants involved in the fieldwork 

d) the data collection methods that were employed 

e) the number and length of the data collection sessions 

f) the time period over which the data were collected. 

In the results this infonnation should be provided by a HDR student to convey the boundaries of their 
study. 

Confirmability 

Shenton (2004) describes confirmability as ‘the qualitative investigator’s comparable concern to 
objectivity’ (p.72). He further explains that steps must be taken to help ensure as far as possible that 
the findings of the research are the result of the ideas and experiences of the participants rather than 
the characteristics and preferences of the researcher. Miles and Huberman (1994) contend that a 
crucial measure for confirmability is the extent to which a researcher admits his or her own 
predispositions. Shenton (2004) adds that recognition of shortcomings in study’s methods and their 
potential effects on the research as well as the inclusion of an in-depth methodological description can 
allow for the integrity of research results to be scrutinised. For a HDR student, this can include 
outlining the limitations of the study and the researcher’s particular frame of reference. For example, a 
classroom teacher conducting research in a classroom may have a different interpretation of the data 
than an instructional designer. 


The Research Design 
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The criteria to ensure reliability of the research design are outlined below. 

Micro phases 

The use of micro phases or prototyping phases in design research is a strategy to ensure reliability of 
the design before the final field work study. As design research aims to ascertain if and why a 
particular intervention works in a certain context, micro research phases provide researchers with an 
opportunity to refine fhe design and fo gain a more informed undersfanding of why an invention may 
(or may nof) work in fhaf contexf (Plomp, 2007). Micro phases involve a series of small scale design 
studies that result in the subsequent revaluation of the materials before the final producf is used in a 
school-based sfudy. The use of micro phases is part of whaf Plomp (2007) refers fo as fhe profofyping 
sfage: ‘each cycle in fhe sfudy is a piece of research in ifself (i.e. having ifs research or evaluation 
question fo be addressed wifh a proper research design)’ (p. 25). Each phase should be presenfed as a 
separafe study as there may be different research questions, population groups, data samples and 
methods of data analysis. This approach was used by Maflimiko (2006), wbo undertook a micro-scale 
investigation of improving the chemistry curriculum in Tanzania, and Squire (2004), who conducted 
three ‘cases’ in the use of the computer game Civilization 111 wifh differenl sfudenf groups in differenl 
settings in order fo refine his design. Figure 2 shows fhe progression fhrough Mafumiko’s (2006) 
sfudy and highlighfs fhe incremenfal progression fhrough fhe phases. If is evidenf fhaf fhere were four 
versions of fhe design prior fo fhe final field fesf in fhe school-based sfudy. Here, fbe design was 
scrutinised by experts and feacbers fo improve fhe maferials used in fhe final study. 


Development of prototypes 


Summative 

evaluation 




Figure 2. Example of research design showing micro phases adapted from Mafumiko (2006) 
in Plomp (2007) 

One of the issues that can arise in design research by a solo investigator is the occurrence of 
conflicting researcher roles; that of the designer and developer, the facilitator and the evaluator of 
research. While playing multiple roles can be beneficial in fhaf a researcher can understand the whole 
process, there are, at times, tensions between the roles. Hence, it is proposed that it is necessary for 
solo design researchers to implement checkpoints during the process to ensure that objectivity is 
maintained. 
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Expert groups 

The inclusion of several expert groups throughout the study to evaluate the materials and data 
collection instruments (surveys, pre-and post-tests and observation schedules) and interrogate the 
findings provides a degree of rigour that may otherwise escape a solo researcher. As the designer and 
developer, solo researchers need to ensure that the instruments and materials are testing what they 
were meant to. This process of external review should occur prior to the initial data collection and any 
school-hased field test. Both Mafumiko (2006) and Squire (2004) subjected their designs to the 
scrutiny of experts. Where possible, the data should be coded by a second researcher. The results 
should also be submitted as conference papers in a timely manner to maintain objectivity via a blind 
peer review process. As the designer and developer of the materials, this process of external and 
internal review maintains the integrity of the research. 

Diverse participant groups 

The use of prototype phases with a range of relevant participants groups is also recommended for two 
reasons. As Reimann (2010) explained, it is often difficult for ethical and practical reasons to conduct 
lengthy studies in classroom situations. For ethical reasons, school student populations are difficult to 
access. School-based studies are also resource-intensive and running ongoing studies with student 
groups can be difficult for a solo researcher. Issues of accessibility can be mitigated by using a range 
of participant groups in the prototype phases. For example. Squire (2004) accessed participants in 
after school programs as well as classrooms. Moreover, by accessing a range of relevant participant 
groups, such as teachers and pre-service teachers, value can be added to a study. By conducting 
studies with a range of participant groups, the materials can be critically analysed prior to the final 
field test with a student group. 

Flexibly adaptive research design 

The nature of design research necessitates adaptability on behalf of the researcher. Adaptability, 
according to Plomp (2007), can be ensured by the researcher being prepared to take on the role of 
designer, advisor and facilitator without losing sight of being a researcher. Plomp (2007) also explains 
that as the research takes place in a real world setting, often the wishes and needs of partners may 
influence the study. This influence may be more prominent in a school-based study that has 
curriculum and ethical requirements. Given that design research takes place in a ‘real world’ context 
and is based on iterative cycles of design and re-design resulting in ongoing changes, it is necessary to 
implement a planning framework. The notion of evolutionary planning is described by McKenney et 
al. (2006) as a planning framework that is ‘responsive to field data and experiences as acceptable 
moments during the course of a study’ (p.84). 


Discussion of small-scale interventions and design research 

In a small-scale study, HDR students often hesitate to generalise their research findings wifhin a 
larger educational context as they are often cautioned against overstating their findings. As Sfake 
(1995) explains: 

It is not uncommon for case study researchers to make assertions on a relatively small 
database, invoking the privilege and responsibility of interpretation. To draw so much 
attention to interpretation may be a mistake, suggesting that case study work hastens to 
conclusions. Good case study is patient, reflective, willing to see another view of the case. An 
ethic of caution is not contradictory to an ethic of interpretation (p.l2). 

Hence, HDR students often make what Stake (1995) in the text The art of case-study research called 
‘petite generalisations’, whereby generalisations are made within the location of the case study. 
Mafumiko (2006) found that using design research allows for the realisation of promising small-scale 
examples of interventions and the generation of methodological guidelines for the design and 
evaluation of such interventions. In this respect, it has been argued that it is through generalisations 
that patterns and their theoretical utility emerge (Barab & Squire, 2004). Moreover, it has been put 
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forward that in design research the cycle of iteration and evaluation of the design may actually reduce 
the overstating of assertions and conclusions (Barah & Squire, 2004). 

As a solo researcher, managing the large data sets that accompany each micro phase can lead 
to selection bias. In a thesis, large quantities of data remain unreported or excluded from the results, 
not because they lacked value, but simply due to the practicalities of a thesis, such as word count and 
the time taken to analyse large data sets. The presentation of observations and findings may be 
compromised by individual biases and experiences regardless of the number of check points set in 
place to maintain integrity. As Stake (1995) further cautions: 

We do not have adequate guides for transforming observations into assertions - yet people 
regularly do it ... the logical path to assertions often is apparent neither to reader nor to 
researchers themselves ... for assertions, we draw from understandings deep within us, 
understandings whose derivation may be some hidden mix of personal experience, 
scholarship, assertions of other researchers. It will be helpful to the reader when such leaps to 
conclusions are labelled as speculation or theory, but researchers often do not (pp. 9-12). 

To reduce the incidence of selection bias the use of criteria, such as those outline in Table 1, may 
assist in ensure that the results presented by a solo HDR researcher using a design research approach 
are reliably and valid. On the whole, as Cobb et al. (2003) explain, the crucial determinant in a solo or 
team research study is that the individual or team has the expertise and skills to develop the initial 
design, undertake the experiment and undertake a systematic and retrospective analysis of the data. 


Conclusions 


One final note on design research and the criteria presented in this paper is that, as explained in a 
range of papers on design research, the large data sets and the undertaking of research in an authentic 
context present a researcher or team or researchers with a range of challenges. These challenges may 
result from gaining access to classrooms, selection bias, analysing and reporting on mixed data sets. 
HDR students often lack the resources to conduct large scale research studies and, consequently, 
focus on micro studies, which may be more manageable and achievable. This does not reduce or 
nullify the value of these contributions to the field. However, what it does suggest is that the 
contributions need to be appreciated on the basis of the trustworthiness of the design and the 
contributions that these studies make to local educational contexts and theory building. 
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